Storefront device, storefront management method, and program

ABSTRACT

First position information indicating positions of approaching people nearing merchandise is acquired. Second position information indicating a position of a subject person who has stretched an arm out towards the merchandise among the approaching people, is detected. A movement action performed on the merchandise is detected. An ID of a person corresponding to the subject person who performed the detected movement action is specified based on a positional relationship between the first position information and the second position information.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is continuation of U.S. application Ser. No.16/640,283, filed on Feb. 19, 2020, which is a National Stage ofInternational Application No. PCT/JP2018/009452 tiled Mar. 12, 2018,claiming priority based on Japanese Patent Application No. 2017-162609,filed Aug. 25, 2017, the disclosures of which are incorporated herein byreference.

TECHNICAL FIELD

The present invention relates to a storefront device, a storefrontmanagement method, and a program.

BACKGROUND ART

Technology is sought for managing automatic payments for merchandisethat a customer wishes to purchase in a storefront. Patent Document 1discloses, as related art, technology relating to an unmannedstorefront.

CITATION LIST Patent Literature [Patent Document 1]

Japanese Unexamined Patent Application, First Publication No. 11-25337

SUMMARY OF INVENTION Problem to be Solved by the Invention

In information processing devices that are installed in storefronts orthe like, it is necessary to improve the merchandise recognitionprecision of merchandise that has been taken in the hand of a customeror that has been returned to a merchandise shelf.

Thus, an objective of the present invention is to provide a storefrontdevice, a storefront management method and a program that can solve theabove-mentioned problem.

Means for Solving the Problem

According to a first embodiment of the present invention, a storefrontdevice includes: a first position information acquisition unitconfigured to acquire first position information indicating positions ofapproaching people who are nearing merchandise; a second positioninformation acquisition unit configured to detect second positioninformation indicating a position of a subject person who has stretchedan arm out towards the merchandise among the approaching people; anaction detection unit configured to detect a movement action performedon the merchandise; and a person specifying unit configured to specify,based on a positional relationship between the first positioninformation and the second position information, identificationinformation corresponding to the subject person who performed thedetected movement action.

According to a second embodiment of the present invention, a storefrontmanagement method includes: acquiring first position informationindicating positions of approaching people nearing merchandise;detecting second position information indicating a position of a subjectperson who has stretched an arm out towards the merchandise among theapproaching people; detecting a movement action performed on themerchandise; and specifying, based on a positional relationship betweenthe first position information and the second position information,identification information corresponding to the subject person whoperformed the detected movement action.

According to a third embodiment of the present invention, a programcauses a computer of a storefront device to execute processes. Theprocesses includes: acquiring first position information indicatingpositions of approaching people nearing merchandise; detecting secondposition information indicating a position of a subject person who hasstretched an arm out towards the merchandise among the approachingpeople; detecting a movement action performed on the merchandise; andspecifying, based on a positional relationship between the firstposition information and the second position information, identificationinformation corresponding to the subject person who performed thedetected movement action.

Advantageous Effects of Invention

According to the present invention, it is possible to improve themerchandise recognition precision of merchandise that has been taken inthe hand of a person in a storefront or that has been returned to amerchandise shelf.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic diagram of a storefront system according to anembodiment of the present invention.

FIG. 2 is a diagram illustrating merchandise shelves and a useracquiring merchandise from a merchandise shelf according to anembodiment of the present invention.

FIG. 3 is a hardware configuration diagram for a storefront deviceaccording to an embodiment of the present invention.

FIG. 4 is a functional block diagram of a storefront device according toan embodiment of the present invention.

FIG. 5 is a diagram illustrating the processing flow in a storefrontdevice according to an embodiment of the present invention.

FIG. 6 is a diagram illustrating the minimum configuration for astorefront device according to an embodiment of the present invention.

DESCRIPTION OF EMBODIMENTS

Hereinafter, a storefront device according to an embodiment of thepresent invention will be explained with reference to the drawings.

FIG. 1 is a schematic diagram of a storefront system provided with thestorefront device according to the present embodiment.

As illustrated in this drawing, the storefront device 1 is communicablyconnected to devices provided in a storefront 20. The storefront 20 isprovided, for example, with an entrance/exit gate 2. A plurality ofmerchandise shelves 21 are provided in the storefront 20. Merchandise isarranged on each merchandise shelf 21.

The storefront 20 that is managed by the storefront system 100 accordingto the present embodiment has a structure in which a user enters orexits the store by passing through an entrance/exit gate 2. It is notnecessary for an employee to always be stationed at the storefront 20.It is also possible for an employee to always be stationed at thestorefront 20. A user takes merchandise from merchandise shelves 21 inthe hand, and exits the store through the entrance/exit gate 2. Untilthe user exits the store through the entrance/exit gate 2, sensingdevices such as image capture devices and motion sensors provided in thestore acquire and transmit, to the storefront device 1, sensinginformation for determining feature information and position informationof the user, and identification information and the positions of themerchandise acquired by the user, or the like. The storefront device 1uses the received sensing information to automatically perform a paymentprocess.

FIG. 2 is a diagram illustrating merchandise shelves and a useracquiring merchandise from a merchandise shelf.

A plurality of first cameras 3 may be provided on each merchandise shelf21. Additionally, a motion sensor 4 for sensing the motion of the useris provided above the merchandise shelf 21. Additionally, second cameras5 for capturing images of merchandise taken in the hand of a user of thestorefront 20 and merchandise returned to the merchandise shelf 21 areprovided above the merchandise shelf 21. The first cameras 3 and thesecond cameras 5 do not need to be provided on the merchandise shelves21. The first cameras 3 and the second cameras 5 may be providedanywhere, such as in the ceiling or in the floor, as long as they arepositions from which it is possible to capture facial images, images ofmerchandise taken in the hand of a user, and images of merchandisereturned to the merchandise shelves 21.

FIG. 3 is a hardware configuration diagram for a storefront device.

As illustrated in FIG. 3, the storefront device 1 is provided, as anexample, with hardware features including a CPU (Central ProcessingUnit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory)103, an HDD (Hard Disk Drive) 104, an interface 105 and a communicationmodule 106. The HDD 104 may be an SSD (Solid State Drive).

FIG. 4 is a functional block diagram of a storefront device.

The CPU 101 of the storefront device 1 reads and executes a storefrontmanagement program that is pre-recorded in a storage unit. As a resultthereof; the storefront device 1 is provided with the functions of acontrol unit 11, a first position information acquisition unit 12, asecond position information acquisition unit 13, an action detectionunit 14, a person specifying unit 15, a sales management unit 16 and aninformation output unit 17.

As illustrated in FIG. 4, the storefront device 1 is connected to adatabase 10. The storefront device 1 is communicably connected, via afirst communication network 8, to the devices provided inside thestorefront 20, such as the entrance/exit gate 2, the first cameras 3,the motion sensors 4, the second cameras 5, merchandise detectionsensors 6, and gate devices 23. The first communication network 8 is,for example, a dedicated communication network for connecting thestorefront device 1 with the devices in the storefront 20. Thestorefront device 1 is also connected, via a second communicationnetwork 9, with a terminal 7 carried by the user of the storefront 20.The second communication network 9 is a mobile telephone network or theinternee.

Due to the above-mentioned functions of the storefront device 1according to the present embodiment, the storefront device 1 acquiresfirst position information indicating the positions of biologicalfeature information of approaching people who are nearing merchandise,based on images obtained from the first cameras 3. The first cameras 3are cameras that are used to detect biological feature information ofpeople appearing within an image capture range, and as an example, areinstalled on the merchandise shelves 21. The first cameras 3 captureimages of the faces of people positioned in front of the merchandiseshelves 21. The biological feature information may, for example, befeature information relating to the face, or feature informationrelating the irises in the eyes. The approaching people may be storeusers, such as customers, or may be a manager who manages the storefront20.

Additionally, the storefront device 1 detects second positioninformation indicating the position of a subject person who has reachedout for merchandise among the approaching people, based on sensinginformation obtained from the motion sensors 4. The motion sensors 4 areinstalled, for example, in the ceiling above the positions of themerchandise shelves 21, and sense a subject person below from theceiling. Furthermore, the storefront device 1 detects merchandisemovement actions. For example, the storefront device 1 detectsmerchandise movement actions based on images from second cameras 5 thatare installed so as to correspond to the respective merchandise shelves21 and that capture images of the merchandise arranged on thosemerchandise shelves 21, and information obtained from the merchandisedetection sensors 6. Merchandise movement actions are actions for takingmerchandise in the hand, actions for returning merchandise to themerchandise shelves 21, or the like. The storefront device 1 specifiesthe biological feature information for a subject person who hasperformed a merchandise movement action based on an ID and the positionof the merchandise on which the movement action was performed, and thepositional relationship between the first position information and thesecond position information. The storefront device 1 acquires the ID ofa person corresponding to that feature information. Then, the storefrontdevice 1 manages sales management information or the like correspondingto the ID of the specified subject person, such as by assigningidentification information for the merchandise on which the movementaction was performed.

The process for detecting merchandise movement actions in the storefrontdevice 1 will he explained.

When a user of the storefront 20 passes through the entrance/exit gate2, code information such as a two-dimensional code image displayed on aterminal 7 is shown to a gate device 23 provided on the entrance/exitgate 2. As a result thereof, the entrance/exit gate 2 reads thetwo-dimensional code image and outputs, to the storefront device 1, anID for a store user or the like indicated by the two-dimensional code.The entrance/exit gate 2 may read the ID of the store user or the likefrom the terminal 7 by means of wireless communication. The storefrontdevice 1 generates sales management information or the like linked tothe ID of a person such as a store user. At this time, merchandiseinformation is not linked to the sales management information. Then, afirst camera 3 captures, from a merchandise shelf 21, images of a personsuch as a store user positioned in front of the merchandise shelf 21,and outputs the captured images or video images to the storefront device1. A motion sensor 4 senses a store user below from above themerchandise shelf 21, such as from the ceiling, and outputs informationobtained by the sensing process to the storefront device 1.

The information sensed and output by the. motion sensor 4 may, forexample, be a range image or the like obtained by converting, to animage, the ranges to the positions of objects, obtained by means ofinfrared rays.

A merchandise detection sensor 6 is installed, for example, for eachitem of merchandise displayed on a merchandise shelf 21. The merchandisedetection sensors 6 may specifically be in the shapes of sheets that arelaid underneath the merchandise, and may be pressure sensors fordetecting pressure due to the weight of the merchandise at each positionset on the sheet, or may be weight sensors for detecting the weightitself. When, for example, a user takes merchandise in his/her hand, amerchandise detection sensor 6 outputs, to the storefront device 1, amerchandise acquisition signal including a sensor ID of that merchandisedetection sensor 6 and coordinates in the storefront of that merchandiseon the merchandise shelf 21. The storefront device 1 specifies apersonal ID on the basis of the correspondence between the times atwhich coordinates indicated by feature information and skeletal frameinformation of people obtained by the first cameras 3, the motionsensors 4 and the second cameras 5 were detected, and merchandise IDs,coordinates and times stored so as to he linked to the sensor IDs of themerchandise detection sensors 6, received from the merchandise detectionsensors 6.

The storefront device 1 links the specified personal ID to themerchandise ID specified by images taken by a second camera 5, andupdates the sales management information and the like recorded in a datatable in the database 10.

When the merchandise is returned to a merchandise shelf 21, themerchandise detection sensor 6 outputs, to the storefront device 1,merchandise return action information including the sensor ID of thatmerchandise detection sensor 6 and position information, such as thecoordinates in the storefront of that merchandise on the merchandiseshelf 21. The storefront device 1 performs a process for unlinking thestore user identification information (ID) and the merchandise IDrecorded in the database 10. Specifically, the storefront device 1performs a process for unlinking the store user identificationinformation (ID) and the merchandise ID on the basis of the merchandiseID and the position information stored so as to be linked to the sensorID of the merchandise detection sensor 6 indicated by that merchandisereturn action information, and the feature information and skeletalflame information of that person.

The storefront device 1 may detect the identification information formerchandise taken in the hand by the user or the coordinates in thestorefront of that merchandise on the merchandise shelf 21 on the basisof images obtained from the second cameras 5 instead of the informationobtained from the merchandise detection sensors 6. Additionally, thestorefront device 1 may detect the identification information ofmerchandise returned to a merchandise shelf 21 and the coordinates inthe storefront of that merchandise on the merchandise shelf 21 based onimages obtained from the second cameras 5. In other words, thestorefront device 1 only needs to detect merchandise movement actionsbased on either the merchandise detection sensors 6 or the imagesobtained from the second cameras 5. A movement action refers to anaction by a user to take merchandise from a merchandise shelf 21, or anaction to return the merchandise to the merchandise shelf 21.

Due to such processes, the storefront device 1 is able to storeinformation regarding which merchandise the user has taken in the handand which merchandise has been returned to the merchandise shelf 21.Additionally, when passing through the entrance/exit gate 2 to exit thestore, the user shows a two-dimensional code image displayed on theterminal 7 to a gate device 23 provided on the entrance/exit gate 2. Asa result thereof, the entrance/exit gate 2 reads the two-dimensionalcode image and outputs, to the storefront device 1, store-exitinformation including the ID of the store user indicated by saidtwo-dimensional code. The storefront device 1 can automatically detectthe merchandise to be purchased by the user on the basis of therelationship between the merchandise ID and the ID of that user recordedas sales management information in the database 10 when the user exitsthe store through the entrance/exit gate 2. The storefront device 1 isable to automatically perform a payment process using the ID of the userrecorded in the database 10, the merchandise ID, and a credit cardnumber or the like to IS be used by the user for payment. Theabove-mentioned processes of the respective devices in the storefront 20and the above-mentioned processes of the storefront device 1 performedfrom when the user enters the storefront 20 until the user leaves aremerely one example. The processes for detecting merchandise purchased bythe user may be performed on the basis of other processes.

As mentioned above, the storefront device 1 detects actions in which astore user takes merchandise from a merchandise shelf 21 and actions inwhich the store user returns the merchandise to a merchandise shelf 21.However, there is a need to raise the recognition precision forrecognizing which person acquired which merchandise. For example, whenthere are multiple people positioned in front of a merchandise shelf 21and one of those people has taken a specific item of merchandise in thehand, the storefront device 1 cannot automatically proceed with amerchandise purchase process unless it is recognized, with highprecision, which person has taken which item of merchandise.Hereinafter, technology for raising the recognition precision forrecognizing which person took which item of merchandise will beexplained. The storefront device 1 may also determine which items ofmerchandise have been acquired by people, such as employees, other thanpeople visiting the store for the purpose of purchasing merchandise.

FIG. 5 is a diagram illustrating the. processing flow in a storefrontdevice.

The first position information acquisition unit 12 in the storefrontdevice 1, as an example, acquires several to several tens of images persecond from each of a plurality of first cameras 3. The plurality offirst cameras 3 are installed in the respective merchandise shelves 21so as to be able to capture images of people positioned in from of theshelves. The first position information acquisition unit 12 detectsbiological feature information for the people appearing in the acquiredimages (step S101). The biological feature information may, for example,be feature information relating to the face, or feature informationrelating to the irises in the eyes. The first position informationacquisition unit 12, upon successfully acquiring biological featureinformation from the images, computes the spatial coordinates at whichthe feature information was successfully detected. For example, supposethat, for the first position information acquisition unit 12,three-dimensional image capture spatial regions are predetermined basedon angles of view and shooting directions.

The first position information acquisition unit 12 acquires, from thedatabase 10, a three-dimensional image capture spatial region for eachfirst camera 3. The first position information acquisition unit 12computes the three-dimensional coordinates, within a three-dimensionalimage capture spatial region, at which feature information appears, bymeans of a prescribed computational expression, on the basis of theacquired three-dimensional image capture spatial region, the coordinateswithin the image of the feature information appearing in the image, thesize of the feature information, and the like. Additionally, the firstposition information acquisition unit 12 uses a conversion expression,for converting coordinates in the three-dimensional image capturespatial region to coordinates indicating storefront spatial regions, tocompute the three-dimensional coordinates, within a storefront spatialregion, at which the feature information appears (step S102).

When biological feature information has been acquired from an image, thefirst position information acquisition unit 12 records, in a firstperson candidate table, in an associated manner, the ID of the firstcamera 3 that transmitted the image, the detection time, the biologicalfeature information, and the three-dimensional coordinates, within astorefront spatial region, at which the feature information appeared(step S103).

The second position information acquisition unit 13, as an example,acquires several to several tens of range images per second from aplurality of motion sensors 4. The plurality of motion sensors 4 areprovided in the ceiling or the like above the merchandise shelves 21,and are installed so as to be able to capture images, facing downwardsfrom above, of people positioned in front of the shelves. The secondposition information acquisition unit 13 analyzes the range images ofpeople appearing in the acquired images, and detects skeletal frameinformation, such as the positions of heads and the axes of armsstretched out by people in the images (step S104). The skeletal frameinformation may include vectors and coordinates indicating straightlines representing arm axes, and the coordinates of the tips of handsobtained by analysis of the range images. The skeletal frame informationincludes at least coordinates or vectors, or expressions representingthe axes of arms for specifying the positions, within coordinates, ofthe head, arms or tips of the hands viewed from above.

When skeletal frame information including an arm axis and a hand tip areacquired from an image, the second position information acquisition unit13 computes the spatial coordinates of the arm or the hand tipindicating the skeletal frame information. As with the first positioninformation acquisition unit 12, the second position informationacquisition unit 13 pre-stores three-dimensional image capture spatialregions based on angles of view and shooting directions. The secondposition information acquisition unit 13 acquires, from the database 10,a three-dimensional image capture spatial region for each motion sensor4. The second position information acquisition unit 13 computes, bymeans of a prescribed computational expression, the three-dimensionalcoordinates, within a three-dimensional image capture spatial region, atwhich skeletal frame information appears, based on the acquiredthree-dimensional image capture spatial region, the coordinates withinthe image of the skeletal frame information appearing in the image, therange from the motion sensor 4, and the like. Additionally, the secondposition information acquisition unit 13 uses a conversion expressionfor converting coordinates in the three-dimensional image capturespatial region to coordinates indicating storefront spatial regions tocompute the three-dimensional coordinates, within a storefront spatialregion, at which the skeletal frame information appears (step S105).

When skeletal frame information has been successfully acquired from animage, the second position information acquisition unit 13 records, in asecond person candidate table, in an associated manner, the ID of themotion sensor 4 that transmitted the image, the detection time, theskeletal frame information, and the three-dimensional coordinates,within a storefront spatial region, at which the skeletal frameinformation appeared (step S106).

The merchandise detection sensor 6, as one example, detects the weightof merchandise or pressure due to said weight. Based on increases,decreases and the like in the weight or pressure, if there is a weightor pressure decrease of a threshold value or more, then the merchandisedetection sensor 6 outputs, to the storefront device 1, merchandiseacquisition action information including a flag indicating a decrease, asensor ID, a merchandise arrangement position (third positioninformation) and an action detection time. Additionally, based on theweight or pressure increases, decreases and the like, if there is aweight or pressure increase of a threshold value or more, then themerchandise detection sensor 6 outputs, to the storefront device 1,merchandise return action information including a flag indicating anincrease, a sensor ID, a merchandise arrangement position (thirdposition information) and an action detection time.

The action detection unit 14 in the storefront device 1 acquiresmerchandise acquisition action information and merchandise return actioninformation from the merchandise detection sensor 6 (step S107). Theaction detection unit 14, based on merchandise acquisition actioninformation, acquires a merchandise ID stored so as to be linked to thesensor ID included in the merchandise acquisition action information. Asa result thereof. the action detection unit 14 detects that themerchandise having that merchandise ID has been acquired from amerchandise shelf 21. Additionally, the action detection unit 14, basedon merchandise return action information, acquires a merchandise IDstored so as to he linked to the sensor ID included in the merchandiseacquisition action information. As a result thereof, the actiondetection unit 14 detects that the merchandise having that merchandiseID has been returned to the merchandise shelf 21 The action detectionunit 14 outputs, to the person specifying unit 15, the merchandise ID,the arrangement position and the action detection time corresponding tothe sensor ID included in the merchandise acquisition action informationor the merchandise return action information.

Upon acquiring, from the action detection unit 14, the merchandise ID,the arrangement position and the action detection time for the movementaction, the person specifying unit 15 performs the determination processdescribed next. In other words, the person specifying unit 15 determineswhether or not the second person candidate table has, recorded therein,skeletal frame information that includes coordinates that are close, towithin a prescribed range, to the three-dimensional coordinates in thestorefront spatial region indicated by the arrangement position, andthat is linked to a detection time within a predetermined timedifference relative to the action detection time. When there is,recorded in the second person candidate table, skeletal frameinformation that includes coordinates close, to within a prescribedrange, to the three-dimensional coordinates of the merchandise on whichthe movement action was performed, and that is linked to a detectiontime within a predetermined time difference relative to the actiondetection time, the person specifying unit 15 acquires that skeletalframe information(step S108).

For example, the skeletal frame information includes thethree-dimensional coordinates of the tip of a hand. When thethree-dimensional coordinates of an item of merchandise and thethree-dimensional coordinates of the tip of the hand approach close toeach other at roughly the same time, the person specifying unit 15infers that the skeletal frame information including thethree-dimensional coordinates of that hand tip is skeletal frameinformation of the person that took the item of merchandise in the hand,and acquires that skeletal frame information. Therefore, based on thethree-dimensional coordinates of an item of merchandise on which amovement action has been performed, the person specifying unit 15acquires the three-dimensional coordinates of a head included in theskeletal frame information acquired as mentioned above, and thedetection time of that skeletal frame information (step S109).

The person specifying unit 15 acquires, from the first person candidatetable, facial feature information that is linked to three-dimensionalcoordinates within a prescribed range from the three-dimensionalcoordinates of the head, and that has a detection time within aprescribed time difference (step S110). The processes in this personspecifying unit 15 are one mode of the processes for specifying theposition (second position information) indicated by skeletal frameinformation of a subject person who has stretched out an arm towards anitem of merchandise based on an acquired merchandise position (thirdposition information), and for detecting the position (first positioninformation) at which facial feature information having a positionalrelationship that is significant based on the position indicated by theskeletal frame information. Facial feature information having asignificant positional relationship refers to facial feature informationthat is, for example, highly associated with or correlated to theposition indicated by the skeletal frame information. It is assumed thatthe storefront device 1 pre-stores, in a personal feature table in thedatabase 10, facial feature information linked to personal IDs. Based onthat stored information, the person specifying unit 15 uses the facialfeature information acquired from the skeletal frame information todetect a personal ID (step S111).

According to the above-mentioned processes, the storefront device 1detects positions based on the motions of the arms or hands of peoplethat have acquired merchandise from merchandise shelves 21 and thepositions of merchandise on which movement actions have been performed.The storefront device 1 determines which merchandise has been acquiredby which person, and which merchandise has been returned by which personbased on the correspondence relationship between the positions based onthe motions of the arms or hands of the people and the positions of themerchandise. Due to such processes, it is possible to determine, moreprecisely than in the conventional art, which merchandise has beenacquired by which person, and which merchandise has been returned bywhich person.

During the processing for acquiring the merchandise acquisition actioninformation and the merchandise return action information indicated bystep S107 above, the storefront device 1 may determine, based on imagesobtained from the second cameras 5, which type of movement action,whether a merchandise acquisition action or a merchandise return action,has been performed. For example, the action detection unit 14 acquires,as one example, about several to several tens of images per second fromeach of the second cameras 5. The second cameras 5 have angles of viewmatched with the ranges of the respective merchandise shelves 21, andcapture images of merchandise placed on said shelves.

Based on image data of each item of merchandise placed on themerchandise shelves 21, the action detection unit 14 sequentiallydetects, by pattern matching or the like, or by movement analysis, thebefore/after movement amounts and presence/absence of merchandise foreach of the items of merchandise appearing in the images, and specifiesthe items of merchandise that have moved. Additionally, for example,when merchandise that was arranged on a merchandise shelf 21 is gone inimages that have been received before and after, it is determined thatthe merchandise has been acquired. On the other hand, when there ismerchandise that was not present arranged on a merchandise shelf 21 inimages that have been received before and after, it is determined thatthe merchandise has been returned to the merchandise shelf 21.

Furthermore, upon determining that merchandise has been acquired, theaction detection unit 14 generates merchandise acquisition actioninformation including the ID of that merchandise and the arrangementposition of the merchandise. The merchandise ID may be an ID that isrecorded so as to be linked to the image data of the merchandise in adatabase 10 or the like. Additionally the merchandise arrangementposition may be the three-dimensional coordinates in a storefrontspatial region computed based on the coordinates in an image captured bya second camera 5. Upon determining that merchandise has been returnedto a merchandise shelf 21, the action detection unit 14 generatesmerchandise return action information including the ID of thatmerchandise and the arrangement position of the merchandise. The actiondetection unit 14 outputs, to the person specifying unit 15, themerchandise ID or arrangement position included in the merchandiseacquisition action information or the merchandise return actioninformation. The subsequent processing by the person specifying unit 15may be a process similar to the above-mentioned person specifyingprocess performed by using the coordinates of merchandise on which themovement actions obtained from the merchandise detection sensors 6 havebeen performed.

As illustrated in FIG. 2, a plurality of second cameras 5 for capturingimages of merchandise may be installed on the merchandise shelves 21,and the action detection unit 14 may determine the merchandise on whichmovement actions have been performed based on images captured by each ofthe second cameras 5. The second cameras 5 may be installed not only onthe merchandise shelves 21, but also in the ceiling or in the floor, Themerchandise recognition precision can be raised by capturing images ofmerchandise on which movement actions have been performed from separatedirections by a plurality of the second cameras 5, and analyzingmultiple images. In this case, the action detection unit 14 sequentiallyacquires, from a plurality of second cameras 5, respective imagescapturing, from different directions, merchandise on which a movementaction has been performed. The action detection unit 14 detects themerchandise in the images acquired by the plurality of second cameras 5by pattern matching. Additionally, the action detection unit 14determines the three-dimensional coordinates of that merchandise bycomputing the three-dimensional coordinates of the merchandise appearingin the images by substituting information such as the shootingdirection, the angle of view and the size of the merchandise into acomputational expression. When the same merchandise appearing in imagesacquired by each of a plurality of second cameras 5 capturing images ofthe same merchandise shelf 21 is detected at the same time, the actiondetection unit 14 recognizes that these are a single item ofmerchandise. Furthermore, the action detection unit 14 recognizes amovement action of the single item of merchandise based on the multipleimages. The database 10 records feature information and imageinformation for the cases in which each item of merchandise is viewedfrom multiple different angles. The action detection unit 14 uses suchmerchandise feature information and image information recorded in thedatabase 10 to recognize merchandise appearing in images newly capturedby the second cameras 5.

Upon detecting the ID of the person who performed the merchandisemovement action in step S111, the person specifying unit 15 outputssales management information to the sales management unit 16. The salesmanagement information includes merchandise acquisition actioninformation or merchandise return action information, which isinformation indicating the ID of that person and the movement action.From the sales management information, the sales management unit 16acquires a personal ID and merchandise acquisition action information ormerchandise return action information. The sales management unit 16determines whether merchandise acquisition action information ormerchandise return action information has been acquired (step S112).When the sales management information includes merchandise acquisitionaction information, the sales management unit 16 performs a purchaseprocess (step S113). In other words, the sales management unit 16 addsone merchandise ID, which is included in the merchandise acquisitionaction information, to merchandise information recorded in a salesmanagement table in the database 10 so as to be linked to the personalID. In this way, it is recorded in the database 10 that the personindicated by the personal ID has purchased the merchandise.

On the other hand, when the sales management information includesmerchandise return action information, the sales management unit 16performs a return process (step S114). In other words, the salesmanagement unit 16 performs a return process that involves deleting onemerchandise ID, which is included in the merchandise return actioninformation, from the merchandise information recorded in the salesmanagement table in the database 10 linked to the personal ID. In thisway, it is recorded in the database 10 that the person indicated by thepersonal ID has removed the merchandise from the items to be purchased.

The sales management unit 16 outputs, to the information output unit 17,the personal ID and a sales management information change notificationindicating that the sales management table has been updated. Uponreceiving the sales management information change notification, theinformation output unit 17, on the basis of that notification, acquiresthe terminal ID of a terminal 7 recorded in the person management table,in the database 10, linked to that personal ID. Based on the terminalID, the information output unit 17 generates sales managementinformation to be transmitted to the terminal 7 (step S115). The salesmanagement information, as one example, may be information including apersonal ID, a list of the names, IDs and the like, the number of eachitem of merchandise, the unit price of each item of merchandise, ofmerchandise taken in the hand by the person specified by that personalID and determined as being merchandise to be purchased, and the totalmonetary amount for all merchandise determined as being merchandise tobe purchased.

The information output unit 17 transmits the generated sales managementinformation to the terminal 7 on the basis of the terminal ID (stepS116). The terminal ID may be a network address of the terminal 7, an IDassigned to a dedicated application program stored in the terminal 7, orthe like. The terminal 7 receives the sales management information andoutputs it to a screen. As a result thereof, sales managementinformation is displayed on the terminal 7 held by the person detectedin step S111, and that person is able to see a list of merchandise thatis to be purchased by that person in the storefront and the totalmonetary amount thereof.

The control unit 11 in the storefront device 1 determines whether or notthe process is to be ended (step S117). When the process is not to beended (NO in step S117), the control unit 11 repeats the process fromstep S101. The processing units in the storefront device 1 perform theabove-mentioned processes in parallel for each person, based oninformation obtained from the sensors provided in the storefront.

In the above-mentioned processes, the sales management unit 16 in thestorefront device 1 performs a process for assigning, to salesmanagement information corresponding to the ID of a person specified bythe person specifying unit 15, the IDs of merchandise on which movementactions have been performed by that person. However, instead of storingsales management information indicating that the merchandise is to bepurchased, the sales management unit 16 may record the merchandise IDinformation in another data table as merchandise value managementinformation indicating that the person has expressed an interest in thatmerchandise.

Additionally, in the above-mentioned processes, the sales managementunit 16 in the storefront device 1 performs a process for assigning, tosales management information corresponding to the ID of a personspecified by the person specifying unit 15, the IDs of merchandise onwhich movement actions for return to merchandise shelves 21 have beenperformed by that person. However, instead of storing sales managementinformation indicating that the merchandise has been returned, the salesmanagement unit 16 may record the merchandise ID information in anotherdata table as merchandise value management information indicating thatthe person has expressed an interest in, but did not go so far as topurchase, that merchandise.

In the above-mentioned processes in the storefront device 1, a person isspecified and merchandise on which a movement action has been performedis specified when just one person is positioned in front of amerchandise shelf 21 and that person acquires the merchandise or returnsthe merchandise to the merchandise shelf 21. However, even when thereare multiple people in front of the merchandise shelf 21, a similarprocess may be used to determine which of the people performed movementactions on which of the merchandise. In this case, the second positioninformation acquisition unit 13 must detect, using range images acquiredfrom each of the motion sensors 4, the skeletal frame information of thepeople appearing in each range image in a precise manner for eachperson. The second position information acquisition unit 13 performs theskeletal frame information detection process for each person based onthe number of people appearing in a range image such that, the morepeople appear in a range image, the heavier the processing load in thesecond position information acquisition unit 13. However, when there isa large processing capacity for a short time in the processing device 1,the skeletal frame information of people appearing in the range imagecan be detected in a short time. By setting the processing capacity fora short time to be large in the storefront device 1, the detection offeature information for people and the detection of merchandise movementactions can be performed in a short time.

In the above-mentioned processes, it is also possible to acquire imagedata for promotional video images based on the personal ID and themerchandise ID included in sales management information. Based on apersonal ID and a merchandise ID included in sales managementinformation, an information output unit 17 specifies, from among aplurality of promotional video images recorded in the database 10, oneor a plurality of promotional video images regarding that merchandise ormerchandise related to that merchandise. Furthermore, the informationoutput unit 17 acquires image data for the one or the plurality ofpromotional vides images that have been specified.

The information output unit 17 may implement control to output thisimage data to the terminal 7 having the terminal ID specified by thepersonal ID, or to a monitor installed in the storefront 20, on amerchandise shelf 21 near the position of the person indicated by thepersonal ID.

FIG. 6 is a diagram illustrating the minimum configuration of thestorefront device.

The storefront device 1 only needs to be provided with at least thefunctions of the first position information acquisition unit 12, thesecond position information acquisition unit 13, the action detectionunit 14 and the person specifying unit 15.

The first position information acquisition unit 12 is a processing unitfor acquiring first position information indicating the positions ofbiological feature information of approaching people who are nearingmerchandise.

The second position information acquisition unit 13 is a processing unitfor detecting second position information indicating the position of asubject person who has stretched out an arm towards merchandise amongthe approaching people.

The action detection unit 14 is a processing unit for detecting amovement action performed on merchandise.

The person specifying, unit 15 is a processing unit for specifying,based on the positional relationship between the first positioninformation and the second position information, identificationinformation corresponding to the biological feature information of asubject person who has performed a detected movement action.

Each of the above-mentioned devices has a computer system in theinterior thereof. Additionally, the steps in each of the above-mentionedprocesses may be stored, in the form of programs, in computer-readablerecording media, and these programs may be read into and executed by acomputer to perform the above-mentioned processes. In this case,computer-readable recording media refer to magnetic disks, magneto-opticdisks, CD-ROMs, DVD-ROMs, semiconductor memory devices and the like.Additionally, these computer programs may be distributed to computers bymeans of communication lines, and the programs may be executed by thecomputers receiving the distributed programs.

Additionally, the above-mentioned programs may be for realizing some ofthe aforementioned functions.

Furthermore, the aforementioned functions may be implemented byso-called difference files (difference programs) that can be realized bybeing combined with programs that are already recorded on a computersystem.

Some or all of the above-mentioned embodiments could be described as inthe following supplementary notes, but they are not limited to thefollowing supplementary notes.

Supplementary Note 1

A storefront device comprising:

a first position information acquisition unit configured to acquirefirst position information indicating positions of approaching peoplewho are nearing merchandise;

a second position information acquisition unit configured to detectsecond position information indicating a position of a subject personwho has stretched an arm out towards the merchandise among theapproaching people;

an action detection unit configured to detect a movement actionperformed on the merchandise; and

a person specifying unit configured to specify, based on a positionalrelationship between the first position information and the secondposition information, identification information corresponding to thesubject person who performed the detected movement action.

Supplementary Note 2

The storefront device according to Supplementary Note 1, wherein:

the person specifying unit is configured to

-   -   acquire the first position information in which facial feature        information of the approaching people appear, second position        information relating to an axis indicating the arm of the        subject person who has stretched an arm out towards the        merchandise, and third position information of the merchandise        on which the detected movement action was performed;    -   specify the second position information based on the acquired        third position information;    -   detect the first position information having a significant        positional relationship based on the second position        information; and    -   specify identification information of a person corresponding to        the facial feature information at the position indicated by the        detected first position in formation.

Supplementary Note 3

The storefront device according to Supplementary Note 1 or SupplementaryNote 2, further comprising:

a sales management unit configured to assign, to management informationcorresponding to the identification information of the subject personspecified by the person specifying unit, the identification informationof the merchandise on which, as the movement action, an acquisitionaction of the merchandise has been performed.

Supplementary Note 4

The storefront device according to Supplementary Note 3, wherein:

the sales management unit is configured to delete, from the managementinformation, the identification information of the merchandise on which,as the movement action, a return action of the merchandise has beenperformed.

Supplementary Note 5

The storefront device according to any one of Supplementary Notes 1 to4, comprising:

an information output unit configured to output, to the subject person,display information regarding the sales management information of themerchandise, based on the identification information of the subjectperson specified by the person specifying unit, and identificationinformation of the merchandise on which the movement action wasperformed.

Supplementary Note 6

The storefront device according to any one of Supplementary Notes 1 to5, wherein:

the first position information acquisition unit is configured to acquirethe first position information within a three-dimensional space offeature information of the subject person captured by first imagecapture means.

Supplementary Note 7

The storefront device according to any one of Supplementary Notes 1 to6, wherein:

the action detection unit is configured to detect whether the movementaction is a acquisition action or a return action of the merchandisebased on a variation in pressure or weight on a placement surface of themerchandise.

Supplementary Note 8

The storefront device according to any one of Supplementary Notes 1 to6, wherein:

the action detection unit is configured to detect whether the movementaction is a acquisition action or a return action of the merchandisebased on movement information of the merchandise captured by secondimage capture means.

Supplementary Note 9

The storefront device according to any one of Supplementary Notes 1 to6, wherein:

the action detection unit is configured to detect whether the movementaction is a acquisition action or a return action of the merchandisebased on presence or absence of the merchandise captured by second imagecapture means.

Supplementary Note 10

A storefront management method comprising:

acquiring first position information indicating positions of approachingpeople nearing merchandise;

detecting second position information indicating a position of a subjectperson who has stretched an arm out towards the merchandise among theapproaching people;

detecting a movement action performed on the merchandise; and

specifying, based on a positional relationship between the firstposition information and the second position information, identificationinformation corresponding to the subject person who performed thedetected movement action.

Supplementary Note 11

A program for causing a computer of a storefront device to executeprocesses, the processes comprising:

acquiring first position information indicating positions of approachingpeople nearing merchandise;

detecting second position information indicating a position of a subjectperson who has stretched an arm out towards the merchandise among theapproaching people;

detecting a movement action performed on the merchandise; and

specifying, based on a positional relationship between the firstposition information and the second position information, identificationinformation corresponding to the subject person who performed thedetected movement action.

Priority is claimed on Japanese Patent Application No. 2017-162609,filed Aug. 25, 2017, the disclosure of which is incorporated herein byreference.

INDUSTRIAL APPLICABILITY

According to the present invention, it is possible to improve themerchandise recognition precision of merchandise that has been taken inthe hand of a person in a storefront or that has been returned to amerchandise shelf.

REFERENCE SIGNS

-   1 Storefront device-   2 Entrance/exit gate-   3 First camera-   4 Motion sensor-   5 Second camera-   6 Merchandise detection sensor-   7 Terminal-   10 Database-   11 Control unit-   12 First position information acquisition unit-   13 Second position information acquisition unit-   14 Action detection unit-   15 Person specifying unit-   16 Sales management unit-   17 Information output unit-   20 Storefront-   21 Merchandise shelf

1. A storefront device comprising: at least one memory configured tostore instructions; and at least one processor configured to executeinstructions to: acquire first position information indicating positionsof approaching people who are nearing merchandise; detect secondposition information indicating a position of a subject person who hasstretched an arm out towards the merchandise among the approachingpeople; detect third position information indicating a position of themerchandise when the merchandise is picked up; specify, based on thefirst position information, the second position information and thethird position information, identification information of the subjectperson corresponding to facial feature information; specify, based onthe third position information, the merchandise as being picked up bythe subject person; and generate management information by associatingthe identification information and the specified merchandise.
 2. Thestorefront device according to claim 1, wherein: the second positioninformation relates to an axis indicating the arm of the subject personwho has stretched the arm out towards the merchandise.
 3. The storefrontdevice according to claim 1, wherein the at least one processor isfurther configured to execute the instructions to: delete, from themanagement information, the merchandise returned by the subject person.4. The storefront device according to claim 1, wherein the at least oneprocessor is further configured to execute the instructions to: output,to the subject person, display information regarding sales informationof the merchandise, based on the management information corresponding tothe subject person.
 5. The storefront device according to claim 1,wherein the at least one processor is further configured to execute theinstructions to: acquire the first position information within athree-dimensional space of feature information of the subject personcaptured by a first camera.
 6. The storefront device according to claim1, wherein the at least one processor is further configured to executethe instructions to: detect whether a movement action indicates that thesubject person picked up or returned the merchandise based on movementinformation of the merchandise captured by a second camera.
 7. Thestorefront device according to claim 1, wherein the at least oneprocessor is further configured to execute the instructions to: detectwhether a movement action is an acquisition action or a return action ofthe merchandise based on movement information of the merchandise.
 8. Thestorefront device according to claim 1, wherein the at least oneprocessor is further configured to execute instructions to: specify,based on detection time of the second position information and detectiontime of the third position information, the identification information.9. A storefront management method comprising: acquiring first positioninformation indicating positions of approaching people nearingmerchandise; detecting second position information indicating a positionof a subject person who has stretched an arm out towards the merchandiseamong the approaching people; detecting third position informationindicating a position of the merchandise when the merchandise is pickedup; specifying, based on the first position information, the secondposition information and the third position information, identificationinformation of the subject person corresponding to facial featureinformation; specifying, based on the third position information, themerchandise as being picked up by the subject person; and generatingmanagement information by associating the identification information andthe specified merchandise.
 10. A non-transitory computer-readablerecording medium storing a program for causing a computer of astorefront device to execute processes, the processes comprising:acquiring first position information indicating positions of approachingpeople nearing merchandise; detecting second position informationindicating a position of a subject person who has stretched an arm outtowards the merchandise among the approaching people; detecting thirdposition information indicating a position of the merchandise when themerchandise is picked up; specifying, based on the first positioninformation, the second position information and the third positioninformation, identification information of the subject personcorresponding to facial feature information; specifying, based on thethird position information, the merchandise as being picked up by thesubject person; and generating management information by associating theidentification information and the specified merchandise.